Prosodic Properties, Perception, and Brain Activity

This paper investigates how differences in syntactic structure influence the speaker’s prosodic realization of temporarily ambiguous utterances and whether the respective prosodic information guides the listener’s sentence comprehension. Exhaustive acoustic analyses of the speech signals as well as behavioral and event-related brain potential (ERP) data of 56 listeners revealed the following results. 1. As predicted by certain theories of syntax-prosody mapping, syntactic differences led to early characteristic changes in the prosodic pattern. 2. Prosodic differences involved word duration, pause insertion, pitch contours, and the loudness function of the speech signals. 3. The disambiguating prosodic cues were immediately decoded by the listeners and prevented them from initial misanalyses typically observed during reading. 4. The processing of Intonational Phrase (IPh) boundaries was reflected by a specific brain response in the ERP. 5. In the presence of other prosodic cues, pause insertion was completely dispensable for the marking and perception of IPh boundaries. 1. PROSODIC PROPERTIES Speech, in contrast to written text, provides prosodic cues in order to express both linguistic (e.g. syntactic) and non-linguistic (e.g. affective) information. In order to realize a certain prosodic effect (e.g. accentuation, prosodic phrasing, etc.), speakers can use a variety of prosodic parameters such as pause insertion, constituent lengthening, and pitch or loudness variations (Cutler, Dahan & van Donselaar, 1997; Alter, Steinhauer & Friederici, 1998). The listener, on the other hand, has to decode and to integrate these different parameters in order to achieve full comprehension. 1.1. Syntax-prosody mapping With respect to linguistic prosody, theories of syntax-prosody mapping assume a more or less direct relationship between the hierarchical syntactic organization of a sentence and its prosodic realization. According to Jacobs (1993), accent positions in terms of their relative prominence (e.g., the weight of accents distributed across a syntactic structure) are calculable once the syntactic structure is known. This approach predicts early prosodic differences for sentences which are temporarily ambiguous due to different directions of branching. This is the case in an Object-Verb-language (OV) such as German. In structures with verb-final word order, i.e. in most subordinate clauses, the verb takes its arguments from its left (’Anna’ in B), whereas the verb of the main clause takes its arguments from its right (’Anna’ in A). ’Anna’ as indirect object of verb_1: (A) [IPh1 Peter verspricht Anna zu ARBEITEN] Peter promises Anna to work [IPh2 und das Büro zu putzen.] and to clean the office 'Anna' as direct object of verb_2: (B) [IPh1 Peter verspricht #] [IPh2 ANNA zu entlasten] Peter promises # to support Anna [IPh3 und das Büro zu putzen.] and to clean the office Note that both conditions are structurally ambiguous up to the verb_2. In (A), the second verb 'arbeiten'/'to work' is intransitive and NP2 'Anna' is the indirect object of the preceding verb_1'verspricht'/'promises'. Here, the second verb is accented. In (B), by contrast, 'Anna' is demanded as direct object by the subsequent transitive verb_2 'entlasten'/'to support'. The syntactic structure in (B) requires a deeper embedding of the NP2 ‘Anna’ which therefore receives the main accent (marked by small CAPITALS in A/B) when applying the algorithm proposed by Jacobs (1993). This can be illustrated via bracketed metrical grids assigning the highest column of beats (‘*’) to the designated constituents (cf. Figure 1). The highest column indicates the position of the main accent. One advantage of using bracketed metrical grids is the possibility to translate syntactic constituents directly into prosodic domains. Brackets in the metrical grid mark boundaries of Intonational Phrases IPh . The relevant art of A consists of ( ) p ( ) only one IPh, whereas (B) is prosodically restructured with an additional IPh boundary after the first verb. Figure 1: The bracketed metrical grids for the relevant parts of (A) and (B). In (A), the highest column of beats is assigned to the lexically stressed syllable of verb_2 ‘arbeiten’ whereas in (B), Peter verspricht.... * * * ) (* ) * * )) (* * )) ... Anna zu ARBEITEN ]IPh1 [ANNA zu entlasten]IPh2 (A) (B) page 227 ICPhS99 San Francisco page 227 ICPhS99 San Francisco the highest column is assigned to the lexically stressed syllable of NP2 ’Anna’. 1.2. Association with Tonal Sequences According to recent tonal sequence models (Reyelt, Grice, Benzmüller, Mayer, and Batliner 1996 for German), the main accent positions derived from the syntactic structure serve as anchor points for the association of tonal sequences. In intonational languages such as German, accents can be assumed to be realized preferably by tonal/pitch variations. We refer to the German-ToBI system (Reyelt et al 1996) in order to predict the correct tonal sequences: Concerning the conditions (A) and (B), we assume the main accents to be associated with rising tonal sequences of the type L+H*. Note that the L+H*sequence is associated via the metrical grid with the lexically stressed syllable of verb_2 in (A), and of NP2 ‘Anna’ in (B). Furthermore, we expect the IPh boundaries to be marked by boundary tones. Sentence internal boundaries are marked by high boundary tones (H%) and/or durational parameters such as pause insertions ('#'). In condition (A), only one prosodic boundary appears after verb_2 whereas in the condition (B) an additional boundary is expected between the first verb and NP2 ’Anna'. L+H* H% (A) [Peter verspricht Anna zu ARbeiten #] H% L+H* H% (B) [Peter verspricht #][ANna zu entlasten #] To summarize the predictions of syntax-prosody mapping, three predictable prosodic parameters have to be distinguished, namely (1) accent position in terms of relative metrical prominence, (2) accent type in terms of the association of prominence with tonal sequences and (3) boundary marking in terms of the tonal or durational realization of the edges of IPhs. 1.3. Results of the Acoustic Speech Signal Analyses 48 sentences of both conditions (A) and (B) were produced by a female speaker of standard German and recorded in a soundproof chamber. Each digitized speech signal (44.1 kHz/16 bit sampling rate) was analyzed with respect to word and pause durations, pitch contour (fundamental frequency, F0), and loudness function (amplitude squares), and then statistically analyzed in paired ttests or ANOVAs. The data clearly confirm the predictions derived from the models of syntax-prosody mapping. Prosodic differences between conditions A and B occurred long before the structures were disambiguated lexically by the verb_2. 1. The different accent positions, i.e. ’arbeiten’ in (A) versus ’Anna’ in (B), were realized by both local pitch maxima and local loudness maxima (cf. Figure 2). As expected, both parameters revealed significant interactions between condition and accent position (p<0.0001 and p<0.01, respectively). 2. The additional IPh boundary in condition (B) was marked by a significant pause insertion of some 150 ms between verb_1 and ’Anna’ (p<0.0001). Moreover, the sentence initial constituent ’Peter verspricht’ preceding the boundary was considerably lengthened in (B) as compared to (A) (p<0.0001).